{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this notebook, we demonstrate the steps needed create a model for predicting Remaining Useful Life for turbofan engines based on data collected by devices and routed into storage via the IoT Hub. The notebook assumes that you have complete the device data generation steps from the [IoT  Edge for Machine Learning](aka.ms/IoTEdgeMLPaper). The data generated from the devices needs to be in an Azure Storage account blob container in the same Azure Subscription as you will use to create the Azure Machine Learning service workspace using this notebook.  \n",
    "\n",
    "The steps we will complete in this notebooks are:\n",
    "   1. Create a Machine Learning service workspace for managing the experiments, compute, and models for this sample\n",
    "   1. Load training data from Azure Storage\n",
    "   1. Prepare the data for training the model\n",
    "   1. Explore the data \n",
    "   1. Remotely train the model\n",
    "   1. Test the model using test data\n",
    "\n",
    "The intent is not to provide an extensive coverage of machine learning in Azure as that is covered in much depth elsewhere [here for example](https://github.com/Azure/MachineLearningNotebooks), but to demonstrate how machine learning can be used with IoT Edge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup notebook\n",
    "\n",
    "Please ensure that you are running the notebook under the Python 3.6 kernal. Intall fastavro and setup interactive shell to display output nicely.\n",
    "\n",
    ">You may see a warning about Matplotlib building the font cache. You may ignore the warning as it is benign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install fastavro\n",
    "!pip install seaborn\n",
    "\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List versions of azureml modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip list | grep azureml-sdk\n",
    "\n",
    "!pip list | grep azureml-train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade tqdm\n",
    "\n",
    "!pip install --ignore-installed entrypoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Move data files to data directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, glob, shutil\n",
    "\n",
    "if not os.path.exists('./data'):\n",
    "    os.mkdir('./data')\n",
    "\n",
    "for f in glob.glob('./*.txt') + glob.glob('./*.csv'):\n",
    "    shutil.move(f, './data/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set global properties\n",
    "\n",
    "These properties will be used throughout the notebook.\n",
    "   * `AZURE_SUBSCRIPTION_ID` - the Azure subscription containing the storage account where device data has been uploaded. We will create the Machine Learning service workspace (ml workspace) in this subscription.\n",
    "   * `ML_WORKSPACE_NAME`  name to give the ml workspace\n",
    "   * `AZURE_IOT_HUB_NAME` - name of the Azure IoT Hub used in creating the device data using the DeviceHarness.  See [IoT  Edge for Machine Learning](aka.ms/IoTEdgeMLPaper) for details.\n",
    "   * `RESOURCE_GROUP_NAME` - name of the resource group where the IoT Hub exists\n",
    "   * `LOCATION` - the Azure location of the IoT Hub\n",
    "   * `STORAGE_ACCOUNT_NAME` - name of the Azure Storage account where device data was routed via IoT Hub.\n",
    "   * `STORAGE_ACCOUNT_KEY` - access key for the Azure Storage account\n",
    "   * `STORAGE_ACCOUNT_CONTAINER` - name of Azure Storage blob container where device data was routed via IoT Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AZURE_SUBSCRIPTION_ID = ''\n",
    "ML_WORKSPACE_NAME = 'turbofanDemo'\n",
    "AZURE_IOT_HUB_NAME = ''\n",
    "RESOURCE_GROUP_NAME = ''\n",
    "LOCATION = ''\n",
    "STORAGE_ACCOUNT_NAME = ''\n",
    "STORAGE_ACCOUNT_KEY = ''\n",
    "STORAGE_ACCOUNT_CONTAINER = ''\n",
    "\n",
    "if (AZURE_SUBSCRIPTION_ID == ''\n",
    "        or ML_WORKSPACE_NAME == ''\n",
    "        or AZURE_IOT_HUB_NAME == ''\n",
    "        or RESOURCE_GROUP_NAME == ''\n",
    "        or LOCATION == ''\n",
    "        or STORAGE_ACCOUNT_NAME == ''\n",
    "        or STORAGE_ACCOUNT_KEY == ''\n",
    "        or STORAGE_ACCOUNT_CONTAINER == ''):\n",
    "    raise ValueError('All values must be filled in') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a workspace\n",
    "\n",
    "## What is an Azure ML Workspace and Why Do I Need One?\n",
    "\n",
    "An Azure ML workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, operationalization, and the monitoring of operationalized models.\n",
    "\n",
    "In addition to creating the workspace, the cell below writes a file, config.json, to a ./.azureml/config.json, which allows the Workspace object to be reloaded later.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "><font color=gray>_Note: currently Workspaces are supported in the following regions: eastus2, eastus,westcentralus, southeastasia, westeurope, australiaeast, westus2, southcentralus_</font>\n",
    "\n",
    "You may need to authenticate with Azure when running this cell.  If so you will see a message like: \n",
    "\n",
    "```To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXXXXXXXX to authenticate.```\n",
    "\n",
    "If you are logged in with an AAD account you will instead be prompted to allow access to Azure.\n",
    "\n",
    "Once you authenticate, the cell will finish creating the Workspace.\n",
    "\n",
    ">To facilitate rerunning the notebook with the same Workspace, the cell first checks for the presence of a config. If it finds the config, it loads the Workspace from the config instead of creating it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "workspace_name = ML_WORKSPACE_NAME\n",
    "subscription_id = AZURE_SUBSCRIPTION_ID\n",
    "resource_group = RESOURCE_GROUP_NAME\n",
    "location = LOCATION\n",
    "\n",
    "if not os.path.exists('./aml_config'):\n",
    "    os.mkdir('./aml_config')\n",
    "\n",
    "#check to see if the workspace has already been created and persisted\n",
    "if (os.path.exists('./aml_config/.azureml/config.json')):\n",
    "    ws = Workspace.from_config(path='./aml_config')\n",
    "else:\n",
    "    ws = Workspace.create(name=workspace_name,\n",
    "                          subscription_id=subscription_id,\n",
    "                          resource_group=resource_group,\n",
    "                          create_resource_group=True,\n",
    "                          location=location\n",
    "                         )\n",
    "\n",
    "    ws.write_config(path='./aml_config')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workspace details\n",
    "\n",
    "Print the Workspace details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "import pandas as pd\n",
    "from azureml.core import Workspace\n",
    "\n",
    "output = {}\n",
    "output['SDK version'] = azureml.core.VERSION\n",
    "output['Subscription ID'] = ws.subscription_id\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.DataFrame(data=output, index=['']).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download data from storage\n",
    "\n",
    "The first step toward creating a model for RUL is to explore the data and understand its shape. We will download the data for this purpose, realizing that in the case of larger data sets only a sample of the data would be used at this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register storage account\n",
    "\n",
    "The Datastore is a convenient construct associated the Workspace to upload/download data, and interact with it from remote compute targets. Register the Azure Storage account and container where device data was routed by IoT Hub using the information about the storage container provided at the beginning of the notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "\n",
    "ds = Datastore.register_azure_blob_container(workspace=ws,\n",
    "                                             datastore_name='turbofan',\n",
    "                                             container_name=STORAGE_ACCOUNT_CONTAINER,\n",
    "                                             account_name=STORAGE_ACCOUNT_NAME,\n",
    "                                             account_key=STORAGE_ACCOUNT_KEY,\n",
    "                                             create_if_not_exists=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Datastore to download data\n",
    "\n",
    "Use the Datastore to download the files to the local machine. The prefix is the top level path to download, which should be the name of the IoT Hub. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.download(target_path=\"./data/download\", prefix=AZURE_IOT_HUB_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load train data\n",
    "\n",
    "The data we just downloaded represent a series of messages sent by each device stored in [Apache Avro](https://avro.apache.org/docs/current/)(avro) format. We will use the fastavro package to deserialize the records from the avro files.\n",
    "Here is an example deserialized record from the avro file. \n",
    "\n",
    "```json \n",
    "{\n",
    "    \"EnqueuedTimeUtc\": \"2018-12-01T01: 16: 22.0950000Z\",\n",
    "    \"Properties\": {},\n",
    "    \"SystemProperties\": {\n",
    "        \"connectionDeviceId\": \"Client_3\",\n",
    "        \"connectionAuthMethod\": {\n",
    "            \"scope\": \"device\",\n",
    "            \"type\": \"sas\",\n",
    "            \"issuer\": \"iothub\",\n",
    "            \"acceptingIpFilterRule\": null\n",
    "        },\n",
    "        \"connectionDeviceGenerationId\": \"636791290544434625\",\n",
    "        \"contentType\": \"application/json\",\n",
    "        \"contentEncoding\": \"utf-8\",\n",
    "        \"enqueuedTime\": \"2018-12-01T01: 16: 22.0950000Z\"\n",
    "    },\n",
    "    \"Body\": b'{\n",
    "        \"CycleTime\": 1,\n",
    "        \"OperationalSetting1\": -0.0001,\n",
    "        \"OperationalSetting2\": 0.0001,\n",
    "        \"OperationalSetting3\": 100.0,\n",
    "        \"Sensor1\": 518.67,\n",
    "        \"Sensor2\": 642.03,\n",
    "       //Sensor 3-19 ommitted for brevity\n",
    "        \"Sensor20\": 38.99,\n",
    "        \"Sensor21\": 23.296\n",
    "    }\n",
    "}```\n",
    "\n",
    "Taken together the messages represent a time series of data for multiple engines. Each engine is operating normally at the start of each time series, and develops a fault at some point during the series. The fault grows in magnitude until system failure (i.e. the failure point for the engine is the final cycle in the set). The remaining useful life (RUL) is therefore expressed as: \n",
    "\n",
    "$$RUL_{current} = Cycle_{max} - Cycle_{current}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create utils for loading data from avro files\n",
    "\n",
    "Define a set of utility methods for loading the data from the avro files. We use thes utilities to load the locally downloaded data. Later in the notebook, these same utilities will form the basis of data processing for remote training (see **Train regression using Azure AutoMl and remote execution** below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import json\n",
    "import pandas as pd\n",
    "\n",
    "from fastavro import reader\n",
    "from os.path import isfile\n",
    "from multiprocessing.dummy import Pool as ThreadPool \n",
    "\n",
    "# parse connectionDeviceId and return the int part\n",
    "# (e.g. Client_1 becomes 1)\n",
    "def get_unit_num (unit_record):\n",
    "    unit = unit_record[\"connectionDeviceId\"]\n",
    "    return int(unit.split('_')[1])\n",
    "\n",
    "# create data row from avro file record\n",
    "def load_cycle_row(record):\n",
    "    json_body = record[\"Body\"].decode()\n",
    "    row = json.loads(json_body)\n",
    "    row.update({'Unit': get_unit_num(record[\"SystemProperties\"])})\n",
    "    row.update({'QueueTime': pd.to_datetime(record[\"EnqueuedTimeUtc\"])})\n",
    "    return row\n",
    "\n",
    "# add row to data frame\n",
    "def append_df(base_df, append_df):\n",
    "    if(base_df is None):\n",
    "        base_df = pd.DataFrame(append_df)\n",
    "    else:\n",
    "        base_df = base_df.append(append_df, ignore_index=True)\n",
    "    return base_df\n",
    "\n",
    "# sort rows and columns in dataframe\n",
    "def sort_and_index(index_data):\n",
    "    #sort rows and reset index\n",
    "    index_data.sort_values(by=['Unit', 'CycleTime'], inplace=True)\n",
    "    index_data.reset_index(drop=True, inplace=True)\n",
    "    \n",
    "    #fix up column sorting for convenience in notebook\n",
    "    sorted_cols = ([\"Unit\",\"CycleTime\", \"QueueTime\"] \n",
    "                   + [\"OperationalSetting\"+str(i) for i in range(1,4)] \n",
    "                   + [\"Sensor\"+str(i) for i in range(1,22)])\n",
    "\n",
    "    return index_data[sorted_cols]\n",
    "\n",
    "# load data from an avro file and return a dataframe\n",
    "def load_avro_file(avro_file_name):\n",
    "    with open(avro_file_name, 'rb') as fo:\n",
    "        file_df = None\n",
    "        avro_reader = reader(fo)\n",
    "        print (\"load records from file: %s\" % avro_file_name)\n",
    "        for record in avro_reader:\n",
    "            row = load_cycle_row(record)\n",
    "            file_df = append_df(base_df=file_df, append_df=[row])\n",
    "        return file_df\n",
    "\n",
    "# load data from all avro files in given dir \n",
    "def load_avro_directory(avro_dir_name):\n",
    "    lst = glob.iglob(avro_dir_name, recursive=True)\n",
    "    files = [x for x in lst if isfile(x)]\n",
    "    pool = ThreadPool(4)\n",
    "    results = pool.map(load_avro_file, files)\n",
    "    pool.close()\n",
    "    pool.join()\n",
    "\n",
    "    dir_df = None\n",
    "    for df in results:\n",
    "        dir_df = append_df(base_df=dir_df, append_df=df)\n",
    "    print(\"loaded %d records\" % dir_df.shape[0])\n",
    "    return sort_and_index(dir_df)\n",
    "\n",
    "# add max cycle to each row in the data\n",
    "def add_maxcycle(data_frame):\n",
    "    # cleanup column if it already exists\n",
    "    if 'MaxCycle' in data_frame.columns:\n",
    "        data_frame.drop('MaxCycle', axis=1, inplace=True)\n",
    "\n",
    "    total_cycles = data_frame.groupby(['Unit']).agg({'CycleTime' : 'max'}).reset_index()\n",
    "    total_cycles.rename(columns = {'CycleTime' : 'MaxCycle'}, inplace = True)\n",
    "    return data_frame.merge(total_cycles, how = 'left', left_on = 'Unit', right_on = 'Unit')\n",
    "\n",
    "# return a remaining useful life class based on RUL\n",
    "def classify_rul(rul):\n",
    "     if (rul <= 25):\n",
    "          return 'F25'\n",
    "     elif (rul <= 75):\n",
    "          return 'F75'\n",
    "     elif (rul <= 150):\n",
    "          return 'F150'\n",
    "     else:\n",
    "          return 'Full'\n",
    "    \n",
    "# add remaining useful life and remaing useful life class\n",
    "# to each row in the data\n",
    "def add_rul(data_frame):\n",
    "    data_frame = add_maxcycle(data_frame)\n",
    "    \n",
    "    if 'RUL' in data_frame.columns:\n",
    "        data_frame.drop('RUL', axis=1, inplace=True)\n",
    "    data_frame['RUL'] = data_frame.apply(lambda r: int(r['MaxCycle'] - r['CycleTime']), axis = 1)\n",
    "\n",
    "    if 'RulClass' in data_frame.columns:\n",
    "        data_frame.drop('RulClass', axis=1, inplace=True)\n",
    "    data_frame['RulClass'] = data_frame.apply(lambda r: classify_rul(r['RUL']), axis = 1)\n",
    "    \n",
    "    return data_frame\n",
    "\n",
    "def create_training_dataset(local_dataset_file):\n",
    "    features = [\"Sensor\"+str(i) for i in range(1,22)]\n",
    "    train_pd = load_avro_directory('./data/download/**/*')\n",
    "    train_pd = add_rul(train_pd)\n",
    "    train_pd.to_csv(local_dataset_file)\n",
    "    ds.upload_files(files=[local_dataset_file], overwrite=True, show_progress=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use utils to load data from download directory\n",
    "\n",
    "This step will take several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import utils\n",
    "import pandas as pd\n",
    "\n",
    "# check to see if this step has run before and if it has load the data rather than recreating it\n",
    "dataset_file_name = 'WebServiceTrain.csv'\n",
    "local_dataset_file = './data/' + dataset_file_name\n",
    "if (not os.path.exists(local_dataset_file)):\n",
    "    create_training_dataset(local_dataset_file)\n",
    "\n",
    "train_pd = pd.read_csv(local_dataset_file)\n",
    "train_pd.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate remaining useful life and RUL class labels\n",
    "\n",
    "Add RUL for regression training and RulClass for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_pd = utils.add_rul(train_pd)\n",
    "\n",
    "cols = ['Unit', 'CycleTime', 'MaxCycle', 'RUL', 'RulClass']\n",
    "#show first 5 rows\n",
    "train_pd[cols].head(5)\n",
    "\n",
    "#show last 5 rows for engine 3\n",
    "train_pd[train_pd['Unit'] == 3][cols].tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#show the first five rows\n",
    "train_pd.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explore the data\n",
    "\n",
    "Visualize the data to start to get a sense of how features like sensor measurements and operations settings relate to remaining useful life (RUL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sensor readings and RUL\n",
    "\n",
    "Create a scatterplot for each sensor measurement vs. RUL. Notice that some measurements (e.g. sensor 2) seem to be correlated strongly to RUL whereas other measurements (e.g. sensor 1) stay constant throughout the life of the engine.\n",
    "\n",
    "    \n",
    "><font color=gray>_Note: the data is limited to the first 10 engine units for speed of rendering_</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "#select the data to plot\n",
    "plotData = train_pd.query('Unit < 10');\n",
    "\n",
    "sns.set()\n",
    "g = sns.PairGrid(data=plotData,\n",
    "                x_vars = ['RUL'],\n",
    "                y_vars = [\"Sensor\"+str(i) for i in range(1,22)],\n",
    "                hue=\"Unit\",\n",
    "                height=3,\n",
    "                aspect=2.5,\n",
    "                palette=\"Paired\")\n",
    "g = g.map(plt.scatter, alpha=0.3)\n",
    "g = g.set(xlim=(300,0))\n",
    "g = g.add_legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operational settings and RUL\n",
    "\n",
    "Create a scatterplot for each operation setting vs. RUL. Operational settings do not seem to correlate with RUL.\n",
    "    \n",
    "><font color=gray>_Note: the data is limited to the first 10 engine units for speed of rendering_</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import utils\n",
    "\n",
    "plotData = train_pd.query('Unit < 10');\n",
    "sns.set()\n",
    "g = sns.PairGrid(data=plotData,\n",
    "                x_vars = ['RUL'],\n",
    "                y_vars = [\"OperationalSetting\"+str(i) for i in range(1,4)],\n",
    "                hue=\"Unit\",\n",
    "                height=3,\n",
    "                aspect=2.5,\n",
    "                palette=\"Paired\")\n",
    "g = g.map(plt.scatter, alpha=0.3)\n",
    "g = g.set(xlim=(300,0))\n",
    "g = g.add_legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train model using Azure AutoMl and remote execution\n",
    "\n",
    "In this section, we will use the Azure Machine Learning service to build a model to predict remaining useful life."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Create remote compute target\n",
    "\n",
    "Azure ML Managed Compute is a managed service that enables data scientists to train machine learning models on clusters of Azure virtual machines, including VMs with GPU support. This code creates an Azure Managed Compute cluster if it does not already exist in your workspace. \n",
    "\n",
    " **Creation of the cluster takes approximately 5 minutes.** If the cluster is already in the workspace this code uses it and skips the creation process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import AmlCompute\n",
    "from azureml.core.compute import ComputeTarget\n",
    "import os\n",
    "\n",
    "# choose a name for your cluster\n",
    "batchai_cluster_name = 'cpucluster'\n",
    "cluster_min_nodes = 0\n",
    "cluster_max_nodes = 3\n",
    "vm_size = \"STANDARD_DS3_V2\" \n",
    "      \n",
    "cts = ws.compute_targets\n",
    "if batchai_cluster_name in cts:\n",
    "    found = True\n",
    "    print('Found existing compute target...%s' % batchai_cluster_name)\n",
    "    compute_target = cts[batchai_cluster_name]\n",
    "else:\n",
    "    print('creating a new compute target...')\n",
    "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = vm_size, \n",
    "                                                                vm_priority = 'lowpriority', #optional\n",
    "                                                                min_nodes = cluster_min_nodes, \n",
    "                                                                max_nodes = cluster_max_nodes)\n",
    "\n",
    "    # create the cluster\n",
    "    compute_target = ComputeTarget.create(ws, batchai_cluster_name, provisioning_config)\n",
    "    \n",
    "    # can poll for a minimum number of nodes and for a specific timeout. \n",
    "    # if no min node count is provided it will use the scale settings for the cluster\n",
    "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
    "    \n",
    "    # For a more detailed view of current BatchAI cluster status, use the 'status' property    \n",
    "    compute_target.status.serialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate AutoML\n",
    "\n",
    "In the interest of time, the cell below uses a short iteration timeout, **3 min**, and a small number of iterations, **10**. Longer iteration timeouts and a greater number of iterations will yield better results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "#name project folder and experiment\n",
    "experiment_name = 'turbofan-regression-remote'\n",
    "label_column_name = 'RUL'\n",
    "dataset = Dataset.Tabular.from_delimited_files(path=[(ds, '/' + dataset_file_name)], separator=',')\n",
    "train_columns = [\"Sensor\"+str(i) for i in range(1,22)]\n",
    "train_columns.append(label_column_name)\n",
    "dataset = dataset.keep_columns(columns=train_columns)\n",
    "\n",
    "automl_settings = {\n",
    "    \"iteration_timeout_minutes\": 3,\n",
    "    \"iterations\": 10,\n",
    "    \"n_cross_validations\": 10,\n",
    "    \"primary_metric\": 'spearman_correlation',\n",
    "    \"max_cores_per_iteration\": -1,\n",
    "    \"max_concurrent_iterations\": cluster_max_nodes,\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "Automl_config = AutoMLConfig(task = 'regression',\n",
    "                             debug_log = 'auto-regress.log',\n",
    "                             training_data = dataset,\n",
    "                             compute_target=compute_target,\n",
    "                             label_column_name=label_column_name,\n",
    "                             **automl_settings\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the experiment\n",
    "\n",
    "Run the experiment on the remote compute target and show results as the runs execute. Assuming you have kept the auto_ml settings set in the notebook this step will take several minutes. If you have increased the number of iterations of the iteration timeout it will take longer.\n",
    "\n",
    ">Note: unlike other cells, this one is not finished until the \"Status\" in the output below shows \"Completed\". If it shows a failure, you can check the status in the Azure portal (link will be at the bottom of the output) to learn more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.experiment import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "experiment=Experiment(ws, experiment_name)\n",
    "regression_run = experiment.submit(Automl_config, show_output=True)\n",
    "RunDetails(regression_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explore the results\n",
    "\n",
    "Explore the results of the automatic training using the run details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reconstitute a run\n",
    "Given the long running nature of running the experiment the notebook may have been closed or timed out.  In that case, to retrieve the run from the run id set the value of `run_id` to the run_id of the experiment. We use `AutoMLRun` from `azureml.train.automl.run`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.automl.run import AutoMLRun\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core import Workspace\n",
    "\n",
    "run_id = 'AutoML_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\n",
    "\n",
    "if 'regression_run' not in globals():\n",
    "    ws = Workspace.from_config(path='./aml_config')\n",
    "    experiment_name = 'turbofan-regression-remote'\n",
    "    experiment=Experiment(ws, experiment_name)\n",
    "    regression_run = AutoMLRun(experiment = experiment, \n",
    "                               run_id = run_id)\n",
    "\n",
    "regression_run.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve all iterations\n",
    "\n",
    "View the experiment history and see individual metrics for each iteration run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "children = list(regression_run.get_children())\n",
    "metricslist = {}\n",
    "for run in children:\n",
    "    properties = run.get_properties()\n",
    "    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}\n",
    "    metricslist[int(properties['iteration'])] = metrics\n",
    "\n",
    "import pandas as pd\n",
    "rundata = pd.DataFrame(metricslist).sort_index(1)\n",
    "rundata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register the best model \n",
    "\n",
    "Use the `regression_run` object to get the best model and register it into the workspace. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = regression_run.get_output()\n",
    "\n",
    "# register model in workspace\n",
    "description = 'Aml Model ' + regression_run.id[7:15]\n",
    "tags = None\n",
    "regression_run.register_model(description=description, tags=tags)\n",
    "regression_run.model_id # Use this id to deploy the model as a web service in Azure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save model information for deployment\n",
    "\n",
    "Persist the information that we will need to deploy the model in the [turbofan deploy model](./turbofan_deploy_model.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "model_information = {'regressionRunId': regression_run.id, 'modelId': regression_run.model_id, 'experimentName': experiment.name}\n",
    "with open('./aml_config/model_config.json', 'w') as fo:\n",
    "  json.dump(model_information, fo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load test data\n",
    "\n",
    "In the test set, the time series ends some time prior to system failure. The actual\n",
    "remaining useful life (RUL) are given in the RUL_*.txt files.  The data in the RUL files is a single vector where the index corresponds to the unit number of the engine and the value corresponds to the actual RUL at the end of the test.\n",
    "\n",
    "The RUL for a given cycle in the training set is given by adding the RUL at test end (from the RUL vector file) to the maximum cycle in the test data and then subtracting the current cycle:\n",
    "\n",
    "$$RUL_{current} =  RUL_{TestEnd} + Cycle_{max} - Cycle_{current}$$\n",
    "\n",
    "Taking unit number 1 as an example:\n",
    "   * Taking the first value from RUL_FD003.txt gives:       $RUL_{TestEnd} = 44$\n",
    "   * The final(max) cycle value from test_FD003.txt gives:  $Cycle_{max} = 233$\n",
    "   * The values for the first 5 cycles for engine 1 are:\n",
    "\n",
    "|Unit|Cycle|Max Cycle|Test End RUL|Remaining Life|\n",
    "|-----|-----|-----|-----|-----|\n",
    "|1|1|233|44|276|\n",
    "|1|2|233|44|275|\n",
    "|1|3|233|44|274|\n",
    "|1|4|233|44|273|\n",
    "|1|5|233|44|272|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Define some methods for loading from text files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import utils\n",
    "import pandas as pd\n",
    "from os.path import isfile\n",
    "\n",
    "def add_column_names(data_frame):\n",
    "    data_frame.columns = ([\"Unit\",\"CycleTime\"]\n",
    "                          + [\"OperationalSetting\"+str(i) for i in range(1,4)]\n",
    "                          + [\"Sensor\"+str(i) for i in range(1,22)])\n",
    "\n",
    "def read_data_file(full_file_name):\n",
    "    data = pd.read_csv(full_file_name, sep = ' ', header = None)\n",
    "    data.dropna(axis='columns', inplace=True)\n",
    "    return data\n",
    "\n",
    "def load_rul_data(full_file_name):\n",
    "    rul_data = read_data_file(full_file_name)\n",
    "\n",
    "    # add a column for the unit and fill with numbers 1..n where\n",
    "    # n = number of rows of RUL data\n",
    "    rul_data['Unit'] = list(range(1, len(rul_data) + 1))\n",
    "    rul_data.rename(columns = {0 : 'TestEndRUL'}, inplace = True)\n",
    "    return rul_data\n",
    "\n",
    "\n",
    "def load_test_data(test_full_file_name, rul_full_file_name):\n",
    "    data = read_data_file(test_full_file_name)\n",
    "    add_column_names(data)\n",
    "    data = utils.add_maxcycle(data)\n",
    "    \n",
    "    rul_data = load_rul_data(rul_full_file_name)\n",
    "    data = data.merge(rul_data, how = 'left', left_on = 'Unit', right_on = 'Unit')\n",
    "    data['RUL'] = data.apply(lambda r: int(r['MaxCycle'] + r['TestEndRUL'] - r['CycleTime']), axis = 1)\n",
    "    data['RulClass'] = data.apply(lambda r: utils.classify_rul(r['RUL']), axis = 1)\n",
    "\n",
    "    return data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read and process the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = \"FD003\" \n",
    "rul_file_name = 'data/RUL_' + dataset + '.txt'\n",
    "test_file_name = 'data/test_' + dataset + '.txt'\n",
    "\n",
    "test_pd = load_test_data(test_file_name, rul_file_name)\n",
    "test_pd.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serialize test data\n",
    "\n",
    "Save off the data so that we can use it when we test the web service in the [turbofan deploy model](./turbofan_deploy_model.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pd.to_csv('./data/WebServiceTest.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test regression model\n",
    "\n",
    "predict on training and test set and calculate residual values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_features = [\"Sensor\"+str(i) for i in range(1,22)]\n",
    "\n",
    "#reload data in case the kernel died at some point\n",
    "if 'train_pd' not in globals():\n",
    "    train_pd = pd.read_csv(\"data/WebServiceTrain.csv\")\n",
    "\n",
    "#load the values used to train the model\n",
    "X_train = train_pd[selected_features]\n",
    "y_train = train_pd['RUL']\n",
    "\n",
    "#predict and calculate residual values for train\n",
    "y_pred_train = fitted_model.predict(X_train)\n",
    "y_residual_train = y_train - y_pred_train\n",
    "\n",
    "train_pd['predicted'] = y_pred_train; \n",
    "train_pd['residual'] = y_residual_train\n",
    "\n",
    "#load the values from the test set\n",
    "X_test = test_pd[selected_features]\n",
    "y_test = test_pd['RUL']\n",
    "\n",
    "#predict and calculate residual values for test\n",
    "y_pred_test = fitted_model.predict(X_test)\n",
    "y_residual_test = y_test - y_pred_test\n",
    "\n",
    "test_pd['predicted'] = y_pred_test;\n",
    "test_pd['residual'] = y_residual_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_pd[['Unit', 'RUL', 'predicted', 'residual']].head(5)\n",
    "test_pd[['Unit', 'RUL', 'predicted', 'residual']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicted vs. actual\n",
    "\n",
    "Plot the predicted RUL against the actual RUL.  The dashed line represents the ideal model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(nrows=2, sharex=True)\n",
    "fig.set_size_inches(16, 16)\n",
    "\n",
    "font_size = 14\n",
    "\n",
    "g = sns.regplot(y='predicted', x='RUL', data=train_pd, fit_reg=False, ax=ax1)\n",
    "lim_set = g.set(ylim=(0, 500), xlim=(0, 500))\n",
    "plot = g.axes.plot([0, 500], [0, 500], c=\".3\", ls=\"--\");\n",
    "\n",
    "rmse = ax1.text(16,450,'RMSE = {0:.2f}'.format(np.sqrt(mean_squared_error(y_train, y_pred_train))), fontsize = font_size)\n",
    "r2 = ax1.text(16,425,'R2 Score = {0:.2f}'.format(r2_score(y_train, y_pred_train)), fontsize = font_size)\n",
    "\n",
    "g2 = sns.regplot(y='predicted', x='RUL', data=test_pd, fit_reg=False, ax=ax2)\n",
    "lim_set = g2.set(ylim=(0, 500), xlim=(0, 500))\n",
    "plot = g2.axes.plot([0, 500], [0, 500], c=\".3\", ls=\"--\");\n",
    "\n",
    "rmse = ax2.text(16,450,'RMSE = {0:.2f}'.format(np.sqrt(mean_squared_error(y_test, y_pred_test))), fontsize = font_size)\n",
    "r2 = ax2.text(16,425,'R2 Score = {0:.2f}'.format(r2_score(y_test, y_pred_test)), fontsize = font_size)\n",
    "\n",
    "ptitle = ax1.set_title('Train data', size=font_size)\n",
    "xlabel = ax1.set_xlabel('Actual RUL', size=font_size)\n",
    "ylabel = ax1.set_ylabel('Predicted RUL', size=font_size)\n",
    "\n",
    "ptitle = ax2.set_title(\"Test data\", size=font_size)\n",
    "xlabel = ax2.set_xlabel('Actual RUL', size=font_size)\n",
    "ylabel = ax2.set_ylabel('Predicted RUL', size=font_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicted vs. residual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1,ax2) = plt.subplots(nrows=2, sharex=True)\n",
    "fig.set_size_inches(16, 16)\n",
    "\n",
    "font_size = 14\n",
    "\n",
    "g = sns.regplot(y='residual', x='predicted', data=train_pd, fit_reg=False, ax=ax1)\n",
    "lim_set = g.set(ylim=(-350, 350), xlim=(0, 350))\n",
    "plot = g.axes.plot([0, 350], [0, 0], c=\".3\", ls=\"--\");\n",
    "\n",
    "g2 = sns.regplot(y='residual', x='predicted', data=test_pd, fit_reg=False, ax=ax2)\n",
    "lim_set = g2.set(ylim=(-350, 350), xlim=(0, 350))\n",
    "plot = g2.axes.plot([0, 350], [0, 0], c=\".3\", ls=\"--\");\n",
    "\n",
    "ptitle = ax1.set_title('Train data', size=font_size)\n",
    "xlabel = ax1.set_xlabel('Predicted RUL', size=font_size)\n",
    "ylabel = ax1.set_ylabel('Residual', size=font_size)\n",
    "\n",
    "ptitle = ax2.set_title(\"Test data\", size=font_size)\n",
    "xlabel = ax2.set_xlabel('Predicted RUL', size=font_size)\n",
    "ylabel = ax2.set_ylabel('Residual', size=font_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Residual distribution\n",
    "\n",
    "Plot histogram and Q-Q plot for test and train data to check for normal distibution of residuals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "import scipy.stats as stats\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(nrows=2, ncols= 2)\n",
    "fig.set_size_inches(16, 16)\n",
    "\n",
    "g = sns.distplot(train_pd['residual'], ax=ax1[0], kde=False)\n",
    "g = stats.probplot(train_pd['residual'], plot=ax1[1])\n",
    "\n",
    "g2 = sns.distplot(test_pd['residual'], ax=ax2[0], kde=False)\n",
    "g2 = stats.probplot(test_pd['residual'], plot=ax2[1])\n",
    "\n",
    "\n",
    "ptitle = ax1[0].set_title('Residual Histogram Train', size=font_size)\n",
    "xlabel = ax1[0].set_xlabel('Residuals', size=font_size)\n",
    "ptitle = ax1[1].set_title('Q-Q Plot Train Residuals', size=font_size)\n",
    "\n",
    "ptitle = ax2[0].set_title('Residual Histogram Test', size=font_size)\n",
    "xlabel = ax2[0].set_xlabel('Residuals', size=font_size)\n",
    "ptitle = ax2[1].set_title('Q-Q Plot Test Residuals', size=font_size)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "\n",
    "Now that we have a working model we want to deploy it as an Azure IoT Edge module.  The [turbofan deploy model](./02-turbofan_deploy_model.ipynb) walks through the steps to create and Edge module."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
